Skip to content

Fix staging embedding CI failures: stale-only self-healing fixture + org cleanup workflow#2068

Open
gramos-lb wants to merge 2 commits into
developfrom
gerry/sdk-staging-embedding-cleanup
Open

Fix staging embedding CI failures: stale-only self-healing fixture + org cleanup workflow#2068
gramos-lb wants to merge 2 commits into
developfrom
gerry/sdk-staging-embedding-cleanup

Conversation

@gramos-lb

@gramos-lb gramos-lb commented Jul 21, 2026

Copy link
Copy Markdown

Fix the staging CI embedding failures: self-healing fixture + org cleanup workflow

What was broken

The staging org that all SDK CI runs share (...G7FMH) has a hard cap of 10 custom embeddings per organization, and it's currently pinned at that cap by leaked test embeddings. The embedding fixture creates sdk-int-<uuid> embeddings and deletes them in teardown — but cancelled/killed runs (the develop workflow has cancel-in-progress: true) skip teardown and leak them. Since the public Embedding model has no created_at, a leaked embedding is indistinguishable from a live one, and the org never recovers. Result: every embedding fixture errors with Max limit of custom embeddings reached : 10 per organization, on develop and on every PR.

Why the naive fix (delete everything named sdk-int-* when the cap is hit) is wrong: hitting the cap doesn't mean the ten occupants are leaked. Up to 10 matrix jobs run concurrently against this org (5 python versions × integration/data, each with -n 32), so some occupants can be live fixtures another job is mid-test with — and lbox-alignerr's conftest uses the same name prefix. A blind sweep would delete embeddings out from under running tests.

What changed

Timestamped names + stale-only self-healing (tests/embedding_cleanup.py, fixture in tests/conftest.py):

  • New embeddings are named sdk-int-ci-v2-<epoch>-<uuid> — the creation time rides in the name, which is the only way to establish staleness.
  • On a cap error (matched narrowly; any other LabelboxError re-raises immediately), the fixture deletes only custom embeddings in the v2 format older than 12 hours, then retries — up to 3 attempts with jitter. The TTL exceeds GitHub's 6-hour job ceiling, so a live fixture can never be classified stale. Legacy-format and foreign names are never touched automatically.
  • Healthy runs are silent; heal attempts print one diagnostic line with the run/worker ids. If all attempts fail, the error is loud and says whether zero stale candidates were found (meaning live fixtures or legacy leaks hold the cap).
  • Fixture teardown is unchanged.

One-time bootstrap (.github/workflows/staging-embedding-cleanup.yml + tests/scripts/cleanup_staging_embeddings.py):

  • Manual workflow_dispatch only, dry_run defaults to true. Targets legacy sdk-int-<32hex> leaks (which the automated path deliberately won't touch) plus stale-v2, using the same imported constants — no duplicated regex/TTL.
  • Safety rails: client is pinned to the staging endpoints and fails closed unless the effective REST host is api.lb-stage.xyz (checked at construction and again before listing); always prints the full candidate list before acting; real runs exit nonzero if any deletion fails.

Unit coverage: 17 new tests in tests/unit/test_embedding_cleanup.py — name parse/build rejection cases, stale selection, exact call-count oracles for the retry orchestration (no 4th attempt, no side effects on unrelated errors), endpoint wiring, fail-closed host check, dry-run and failure-exit behavior.

Important: this PR's own embedding jobs will still be red

A new workflow can't be dispatched until it exists on the default branch, so the bootstrap can only run after merge — the legacy leaks keep the org pinned until then. Red embedding jobs on this PR are expected and are exactly the pre-existing failures this PR fixes.

Post-merge runbook

  1. Wait for a quiet moment on both staging consumers: Labelbox Python SDK Staging and LBox Develop (lbox-alignerr integration uses the same org and naming; the script can't detect active owners).
  2. Run Staging Embedding Cleanup with dry_run: true; review the candidate list.
  3. Re-run with dry_run: false; a green run means the org is actually clean (any failed deletion exits nonzero).
  4. Re-run the failed jobs on any red PR to confirm the embedding failures clear.

Residual: lbox-alignerr still creates legacy-format names, so a cancelled LBox Develop run can re-pin the org; the fix is re-running this workflow (migrating alignerr to v2 names is a sensible follow-up, deliberately out of scope here).

Note on the second commit

The first push failed CI at collection: CI invokes pytest as a console script (via rye), which — unlike python -m pytest — doesn't put the project directory on sys.path, so the new tests.* import in conftest wasn't resolvable (local validation used python -m, which masked it). Fixed with a guarded sys.path bootstrap in tests/conftest.py right before that import; deliberately not via [tool.pytest.ini_options] pythonpath, which changes pytest's rootdir semantics and breaks the existing from ..conftest relative imports (verified against clean develop).

How to test

  • pytest tests/unit/test_embedding_cleanup.py (17 tests) or the full unit lane (282 passing).
  • python -m tests.scripts.cleanup_staging_embeddings --help from libs/labelbox.
  • Live verification is the post-merge runbook above.

Co-authored-by: Cursor <cursoragent@cursor.com>
CI runs pytest as a console script, which never puts the project directory
on sys.path, so the new 'from tests.embedding_cleanup import ...' failed at
conftest load (ModuleNotFoundError: No module named 'tests'). Local runs
masked this because 'python -m pytest' adds the cwd. Insert the project
directory in conftest before the tests.* import; verified against all three
lanes with the console-script invocation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant